WarpSearch - The Macintosh Web Server File Searching CGI

As noted in MacWorld UK's July '96 Internet CD for WebMasters
and as featured in the May 1997 issue of TidBits


What is WarpSearch?

WarpSearch is a *fast* non-indexed TEXT and HTML search utility that functions in both local and CGI/ACGI modes. It works with any CGI-compatible Macintosh web-server, including NetPresenz, MacHTTP, WebSTAR, Quid Pro Quo, Web Server 4D, and many others. Speed tests on an old Mac IIci measure WarpSearch at over 500kb/sec and the newer PowerPC Macs perform far better. The bigger the files, the faster the search!


Get the current release (2.6) of WarpSearch here, or check out the latest release info (the always-up-to-date version of this page) at the WarpSearch home page!


Some sites running WarpSearch

Southern Hills http://emc2.acu.edu/shsearch.html
Aspen Institute http://www.aspeninst.org/dir/Warpsearch.html
Nu-Metrics http://www.nu-metrics.com/search.html
The Online Daily News http://www.dailynews.net/news/wire/WarpSearch.html
ComTechLab http://comtechlab.iupui.edu/warpsearch.html

Unlike other applications of its type, WarpSearch will perform blazingly fast searches on text files of any size without adjusting its RAM requirements! WarpSearch consumes a mere 450k of RAM. The Webster's Bible (4.4Mb uncompressed) is at this fast Internet site for your own web-site and speed tests.

WarpSearch is $50 per server with reduced rates for clusters and world-wide use, and free upgrades for as long as you own it (and I'm upgrading it). Electronic methods of easy payment are provided through Kagi (details here and in the archive). Upon receipt of payment, instructions will quickly be sent via e-mail to allow you to register your copy.

Author: Glen Stewart (glen_stewart@associate.com)

What do I need to run WarpSearch?

a). Any functioning CGI-capable Macintosh web server

b). Folders full of TEXT and/or HTML files that remote Web-browsers can search

How do I set WarpSearch up and run it?

1. Place WarpSearch and the WarpSearch Helper application wherever your other CGI's are located. Follow your web-server's documentation for using CGI's. The searchable folders scanned by WarpSearch may be aliases and may contain aliases to files and folders on any mounted volume. Obviously, if WarpSearch references a file outside of your server's file delivery area(s), the remote browser won't be able to retrieve it, so think these steps through. (-:

2. Carefully review and modify the included file, "warpsearch.html" to suit your needs. If you want the 33% speed boost of CGI mode during searches, rename WarpSearch.acgi to WarpSearch.cgi and modify the reference to WarpSearch.acgi in the HTML file to: <FORM ACTION=WarpSearch.cgi METHOD=POST>
Note that CGI mode hogs the web server until the search is complete, whereas ACGI mode shares time and allows the web server to service other connections.

3. WarpSearch *must* see the following line (part of a FORM definition) in the HTML file:

<SELECT NAME="searchFolder">

...and that line must be immediately followed by other lines like:

<OPTION VALUE="BBS_MUG_news:MacDeveloper">Macintosh Developer's Mailing List
<OPTION VALUE="Digests">Mailing List Digests
<OPTION VALUE="ministry_files">Ministry Files
<OPTION VALUE="The_Bible">The Bible

Note that sub-folders are allowed as noted in the first line above!  A new feature as of WarpSearch 2.4

The names within the quotes are the folder names containing files you want WarpSearch to look through when local or CGI forms-based search requests are made. These names must *exactly* match the folder names (which may be aliases) where you store your searchable files. The names after the ">" are what your remote web-browser user will see when using the HTML form's pull-down menu. Fire up NetScape or MS Explorer by drag & dropping "warpsearch.html" onto it, and you'll see this menu.

4. The end of your searchable folder list is reached when WarpSearch sees the line:

</SELECT>

While WarpSearch starts up, it scans your "warpsearch.html" file and builds the local search menu based on the folders you have defined in your HTML form. Don't worry - if you mis-spell a folder name, WarpSearch will gracefully avoid using that folder when it performs searches. You can have anything else you want in your HTML file, as long as the above conditions are met and the search text uses the variable "searchText" as shown in the following line:

<INPUT TYPE="text" NAME="searchText" SIZE="25" MAXLENGTH="40">

5. To hide files with certain suffixes, simply add use a HTML definition modeled after <INPUT TYPE="hidden" NAME="filter" VALUE=".bak.fdml.">at the beginning of your form to specify all the file types that should be skipped during a search. Note the trailing period after "fdml" - your last entry must end with a period!

6. Double-click on the WarpSearch application and note the contents of the "Search..." menu. These should be the folder names (which may be aliases) located in the same directory as WarpSearch that contain the files you wish to search. You may select any of these menu items to try a local search now. Fast, huh? The blue dots to the left of each document name may be clicked on to launch that document with its creator. This function requires the WarpSearch Helper in the same folder was WarpSearch.

7. Now run your favorite web-browser and remotely access the "warpsearch.html" form we've discussed so far. Try a WarpSearch! Once registered, remote users will no longer see the UNREGISTERED title bar and memo within the form displaying how many days you've been evaluating WarpSearch.

8. For best speed, leave WarpSearch running all the time, or you can set a timeout value (of 1,5,10,20,40 or Never ) from the File menu, though it only takes a couple milliseconds to start up.

9. Feel free to use the WarpSearch.GIF for your own WarpSearch web page...

10. In the WarpSearch archive's "Cool HTML Examples" folder, there are sample header and footer files that you can place in the same folder as WarpSearch. You can modify them to suit your needs and create any desired effect or follow any page layout standard you follow at your site. This feature is enabled only for registered users.

11. Registered users will be notified of and given new releases via e-mail as enhanced versions of WarpSearch are released. Registered users will also be included in beta releases when available.


Release Notes:

Apr 20, 1996 v1.0 - first release of WarpSearch
developmental v1.1 - Interface upgrade. Fixed bug in 0-byte file search. Modularized code.
developmental v1.2 - Setup file check, concurrent local/remote searches enabled. RAM requirement cut to 300k!
May 9, 1996 v1.3 - Proper HTML encoding of Mac'ish file paths. Second release of WarpSearch.
developmental v1.4 - Major rewrite for speed and alias support for the initial search folder. Added WarpSearch Helper.
June 22, 1996 v1.5 - Added recursive folder search capability and added workaround for a WebSTAR RFC1630 compatibility bug. Third release of WarpSearch.
July 6, 1996 v1.6 - Implemented ACGI mode of operation in addition to the existing CGI mode. Be forwarned that busy servers running in ACGI mode will be slow search engines!
July 15, 1996 v1.7 Added ISINDEX and URL-based searches, hidden file types, and revised the results link-back page to reference the referer rather than WarpSearch.html, allowing multiple search pages across the site.
July 16, 1996 v1.7.1 Oops - CyberDog bit me. The URL search technique wasn't working with other browsers. WarpSearch now requires a + to separate search arguments. ISINDEX does not work with NetScape, Mosaic or Lynx due to their rejection of multiple search arguments, but Microsoft Explorer and CyberDog perform just fine. I'd love to see NetScape fix this! Until then, the ISINDEX method may be impractical <sigh>.
Aug 1, 1996 v1.8 includes these exciting new features:
  1. FULL alias support! Aliases can span any mounted volume at any folder depth. The organizational capabilities this allows are endless.
  2. Multi-System-Folder registration functionality.
  3. A brief warning to the searcher when the search results are truncated at the 32k mark. This is a CGI/AppleEvent limitation.
  4. Uses a modified time-sharing routine to give better search times when the app is in the background in ACGI mode. Long searches will give timeouts w/NetPresenz (and others?), so use this mode carefully.
developmental v1.9 - Revised search-reply URL file paths to work on all types of web servers, regardless of file hierarchy.
Aug 23, 1996 v2.0 - New application icon! Added ability to quit after (1,5,10,20,40, or Never) minutes of CGI inactivity to conserve server RAM. Added custom header and footer capability to search results page (registered version only). Fixed error in handling ISINDEX and URL searches from v.1.71.
Oct 13, 1996 v2.1 - Modification dates of returned files now shown. Added speed summary at end of output for registered version. Enabled header/footer aliases and made handling of broken aliases encountered during a search more robust.
Nov 12, 1996 v2.2 - Tripled speed of WarpSearch in ACGI mode!  Relative to CGI mode, ACGI mode now operates at roughly 75% of top search speed.
Dec 10, 1996 v2.3 - Enhancements/fixes for this release:

1. Internal error logging. On very rare occasions, WarpSearch would put up a dialog if an internal error occured. All such errors are now logged in a text file called "WarpSearch Errors.txt" so they can be sent to me for diagnosis. This file is created in the same folder as WarpSearch. Error dialogs are no longer allowed.

2. Some Mac's resolve their domain name with a "." at the end. This name makes it into the "svnm" (serverName) variable as CGI's handle AppleEvents and would result in an incorrect URL if used. WarpSearch now filters out the trailing period for URL use, if present.

3. WarpSearch was not correctly resolving FILE aliases (folders were fine) to perform content searches. The result was that a folder search would abort when the first file alias was encountered. Fixed.

4. Launching documents from the local-mode search window (by clicking on the blue dots) was off-kilter after the window width was increased a couple versions ago. This is now fixed.

Jan 29, 1997 v2.4 - Enhancements & fixes:
  • Better parsing of the WarpSearch.html file.  This should allow modification of this file using your favorite WYSIWYG HTML processor.
  • The much-requested sub-folder feature (undocumented in releases prior to 1.8) is back and is fully supported!
  • Case-sensitive searches are now provided!  See the included WarpSearch.html file for proper use. WarpSearch defaults to case-insensitive.
  • A "Filename search only" option may also be included in your search forms.  This allows searches by file name rather than content.  See the included WarpSearch.html file for proper use.
  • Mundane fixes for: GET method used by URL and ISINDEX search methods, removal of another error dialog that we never wanted to see, improved handling of garbled search requests, and a 50k increase in memory for WarpSearch.
Feb 24, 1997 v2.5 - This version resolves a problem brought on by System 7.6, so if you are running 7.6, you should install it.

The version also changes the way the "_" underscore character is handled in the search results page - apparently some versions of WebSTAR are still not compliant with RFC1630, so this character is no longer encoded for this reason, as well as because every browser platform handles it unencoded.

Jun 4, 1997 v2.6 - Revised registration mechanism to support sites with multiple system folders.  I've made an attempt to contact all existing owners with updated information on this.

Future Enhancements:

As of v2.0, all requested features have been implemented. If you have specific reasonable needs and are a registered WaprSearch user, please drop me a note describing your feature request.

Warranty Disclaimer and Liability Limitations:

You expressly acknowledge and agree that use of the WarpSearch Software is at your sole risk. The WarpSearch Software is provided "AS IS" and without warranty of any kind.

UNDER NO CIRCUMSTANCES, INCLUDING NEGLIGENCE, SHALL THE AUTHOR BE LIABLE FOR ANY INCIDENTAL, SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES ARISING OUT OF OR RELATING TO THIS SOFTWARE. In no event shall The Author's total liability to you for all damages exceed the amount paid for the WarpSearch Software.